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The Use of Occupational Licensure Examination Results in Outcomes Assessment 



Abstract 

The movement toward greater accountability in higher education 
has made it necessary for institutions to develop extensive 
assessment plans to measure student outcomes. A performance 
indicator available to institutions are occupational licensure 
examination results of their graduates. These results used in 
conjunction with other student outcomes data can be a very useful 
tool in an institution's assessment effort. Discussion will include 
acquiring examination feedback, using the data, and the limitations 
of this outcomes measurement. Persons interested in student and 
program assessment are the intended audience for this presentation. 

The purpose of this study is threefold. The validity of licensure examination results as 
a performance indicator will first be examined. Second, the compiled information regarding 
sources for acquiring examination feedback will be analyzed. Third, the uses and limitations of 
the feedback will then be discussed. Persons interested in assessment may find this information 
useful for understanding what data are available to departments and how scores can besi be used 
in student and program assessment. 

When the ultimate objective in a degree field is preparation for entry into the job market, 
it is important that the outcomes instrument measures occupational readiness (Nichols, 1991). 
According ro the literature, licensure examination results are recognized as a potentially valid 
tool in major area student outcomes assessment. For example, James O. Nichols in A 
Practitioner's Handbook for Irstructional Effectiveness and Student Outcomes Assessment 
Implementation (1991) stated that licensure examinations are "among the best end-of-program 
assessment procedures" (p.36). In addition, Sarah M. Dinham, in her study of assessment, 
accreditation and licensure (1989) concluded that "professional certification/licensure presents 



1 



attractive possibilities for broadening the assessment data base 11 (p.4). 

Licensure examinations are widely used for protecting the health and welfare of the 
general public. "A license confers on its holder the right to use a title and to provide certain 
services that the law makes illegal for non-licensed persons to provide" (Kane, 1986, p. 146). 
Colleges and universities were affected when the Manpower Acts required that licensure in 
certain professional occupations be coupled with graduation from an accredited university 
program (Lenn, 1987). The exams serve as a uniform measurement device that attempts to 
maintain curriculum consistency among academic programs between institutions, and 
consequently licensing boards have a high degree of control over program length and curriculum 
content (Kane, 1986). 

Licensure is a governmental function, where certification is a function of professional 
organizations. Some medical fields licensure is not required, therefore certification is used as 
a means of recognizing persons who have obtained a predetermined level of competence in the 
given field (Lenn, 1987). These certification exams have been found to be widely used as 
assessment tools in place of a licensure exam. Therefore, a sample of these fields are included 
on the attached summary sheet (appendix 1). However, in some fields, for example nursing, 
certification examinations are used to recognize training beyond that of entry-level. After a 
graduate has passed the initial licensure exam, they may choose to specialize in a certain area 
within the field. The latter type of certification examinations are not included in this study. 

According to Bogue and Saunders (1992), professional degree programs should remain 
closely attuned to skills necessary for entry into the profession. "To do otherwise would be 
institutional suicide, for no institution can long survive without a reasonably high proportion of 
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its graduates meeting professional licensure requirements and, thus becoming licensed to 
practice" (p. 122). It was warned by the State Higher Education Executive Officers (1987) in 
their policy statement on program and institutional assessment, that "if an institution offers 
academic programs designed to lead to licensure or certification, and graduating students then 
have difficulty entering the profession, these programs need careful attention" (p.3). In addition, 
bodies such as licensing boards, state legislators and senior administrative officers often base the 
quality of both graduate and undergraduate professional programs on licensure examination pass 
rates of its students (Applebaum, 1988). 

Using Licensure Examination Scores 

Licensure examination scores are available to departments for a variety of disciplines 
including accounting, architecture, engineering, nursing, law and numerous medical professions. 
Most academic departments automatically receive feedback regarding their graduate's 
performance from their respective licensure board. Analysis of this feedback is performed best 
within the department because knowledge of curriculum, trends, and the profession is strongest 
on that level. In addition, using examination results in major area assessment "... will have 
obvious leverage on the department's faculty. . . 'batting averages' are often developed from 
such results" (Harris, 1985 p.20). 

Even though departments wish to compare its student's performance with those of 
selected peer institutions, information gathered from the literature and licensure boards shows 
that examination feedback rarely includes comparative data other than national norms and 
averages. However, the nursing and pharmacy boards include state-wide averages in the 
feedback to institutions. The state bar associations release only institutional scores to the law 



school deans, however the BAR/BRI digest publishes both the state-wide and nation-wide 
averages. The most detailed report on licensure exam results is published by the National 
Association of State Boards of Accountancy. A publication is released annually with data for 
each institution whose graduates completed the examination. The data are broken down by first- 
time and repeat-takers, and by level of college degree. This publication makes it easy for an 
institution to compare its graduate's performance with those from other institutions. The attached 
documentation (appendix 1) outlines the name and level of the exam, whether it leads to 
licensure or certification, the amount of feedback that can be obtained from the licensure boards, 
and what steps are needed (if any) to acquire data. Most boards automatically send test results 
to the institution in which an examines graduated. The architecture, landscape architecture, and 
psychology boards, however, require a data request from the dean or director of the department. 

There are two main functions of licensure examination results in assessment: student 
outcomes assessment and program assessment. The quality of feedback from licensure boards 
may dictate which function the results will best serve. If the results are reported without student 
identifiers, departments may be limited to using the results strictly for program assessment. 
Whereas if the feedback includes individual scores with identifiers, the scores can then be 
coupled with other measures for student outcomes assessment. 

Program Assessment 

Using annual feedback to develop institutional and national trends over time allows an 
institution to monitor the level of change in exam scores. These trends can then serve as a 
mechanism for curriculum design and reform (Gilley & Galbraith, 1986) and for evaluating 
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program effectiveness (Clageti, 1989). 

Not every licensure examination is a good instrument to measure program effectiveness. 
When there is a high degree of overlap between the content of the curriculum and the 
examination, overall and sub-topic scores can be used to identify program strengths and 
weaknesses (Kane, 1986). Sample copies of the examination should be reviewed by the 
department to determine to what extent the discipline's curriculum is being measured. Licensure 
examinations are usually weighted toward testing only those skills pertaining to public safety and 
may sometimes exclude other skills. When the exam is only pertinent to a small part of a 
curriculum, additional measures are necessary for an overall assessment effort (Dinham, 1989). 
For example, some teachers' examinations cover a broad knowledge base including more 
knowledge gained from other disciplines than from the education curriculum itself. Therefore, 
this type of exam would not be a valid instrument to determine program effectiveness. Other 
exams such as engineering and architecture tend to measure very specific technical abilities as 
opposed to general knowledge. However, caution should be taken to ensure that results from 
specialized examinations, such as civil engineering are compared with the only the respective 
program as opposed to the entire engineering program. 

When institutions receive licensure exam feedback, designation is usually made between 
the first-attempt pass rate and the subsequent-attempt pass rate. Examination feedback will 
usually be formatted with scores of first-time takers separated from those of repeat takers. The 
average number of attempts can to be monitored to track the success of repeat-takers, especially 
in disciplines where multiple attempts are common (Clagett, 1988). For example, the 
architectural registration examinees average three or more attempts to pass all nine sections 
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(Oklahoma State Board of Governors, 1993), where many other examinations may average one 
to two attempts. National average scores and average number of attempts required to pass an 
exam are available from examination feedback or from the governing boards administering the 
examination. 

Student Outcomes Assessment 

When individual student scores are available with identifiers, licensure examination 
results can be used as a component of multiple measures to assess student learning. The scores, 
coupled with employer surveys, job placement information and other end-of-program 
measurements provide a broad view of the major area student outcomes (Dinham, 1989). 

Many institutions use their pass rates in conjunction with other measures of student 
outcomes such as: clinical or practicum experiences, pre-graduation exam results, capstone 
courses, final grade-point average, fieldwork and job placement. For example, a nursing 
program reported that scores from the nurr.ing licensure examination are correlated with the 
individual student's performance on the National League of Nursing (NLN) exams taken during 
the junior and senior years, final grade-point-average, clinical performance, and gender. These 
results are then compiled for use in program assessment and student success prediction. 



Limita t ions 

Some precautions should be taken when using the scores for assessment purposes. First, 
as discussed earlier, an examination may not be an effective outcomes instrument if it does not 
resemble the curriculum of the discipline being assessed. Second, the variation of feedback from 
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respective licensure boards in regards to the use of student identifiers is a obstacle for student 
outcomes assessment. If individual scores do not have identifiers, they can not be paired with 
other measures of student outcomes for a comprehensive assessment plan. Third, reliance on 
self-reported scores when feedback is not available through licensure boards, introduces 
additional problems. Fourth, required internships in some disciplines, that occur between the 
completion of a degree and the examination prevents a direct linkage between curriculum quality 
and a test score (Nichols, 1991; Dinham, 1989). Finally, not all graduates of professional 
disciplines choose to take licensure examinations and instead decide to work in related fields or 
specialize in a non-licensed area (Dinham, 1989). 

Some disciplines have the advantage of receiving individual student scores with identifiers 
whereas other disciplines have to be satisfied with receiving only aggregated ^^.titutional data. 
Many states have privacy laws which prohibit licensure boards from sending scores with 
identifiable variables such as name, gender, ethnic background, and year of graduation. Most 
licensure boards will report institutional results in aggregated form to remain within the limits 
of these laws. However, boards are permitted to have the examinees sign releases to allow the 
release of their scores to their graduating institution (Korb, 1992). When a researcher is limited 
to using aggregated institutional data without knowing the population, they are limited to using 
trend analysis for program assessment. 

When feedback is not available through licensure boards, and an institution is limited to 
using self-reported scores, the results may not be representative of the group. If the data is being 
gathered through a survey instrument, results may be biased and unreliable because of low 
response rates. Those who returned the surveys are more likely to be the ones who have found 
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jobs or have passed the examinations (Korb, 1992). 

A direct linkage between curriculum and program quality and a test score may not exist 
when the licensure examination is taken after a lengthy internship. Experiences during the 
internship often build upon the student's existing knowledge base that was formed during the 
completion of the degree program. Therefore, examinations taken during the final year of 
college or immediately following graduation are more accurate measure of program quality than 
those taken after an internship (Clagett, 1989). 

Many graduates upon completion of a program may choose to work in a related field or 
continue on to graduate work, therefore scores will not be available for all graduates. In 
addition, in some fields it is optional to become license^ and some choose not to take the exam. 

Discussion 

Examination results may also be used in institutional assessment. The State Higher 
Education Executive Officers (1987) stated in their policy statement on program and institutional 
assessment that "the performance of students on licensure and certification examinations should 
be used as an appropriate measure forjudging program and institutional quality' 1 (p.3.). As a 
measure of institutional quality, The Student-Right-to-Know and Campus Security Act of 1990 
(SRK) called for a feasibility study regarding the release of additional measures of student 
outcomes including institutional licensure exam success rates (Legislative History, P.L. 101- 
542). The proposed addition to the act called for the reporting of a single institutional pass-rate 
on licensure examinations. 
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The SRK feasibility study completed a comprehensive look at licensure examinations and 
found that a single institutional pass rate would be impossible. Problems occur when one 
attempts to combine pass rates of licensure examinations for all disciplines in one single 
institutional pass rate. Because of the number of examinations, the variability of the feedback 
from licensure boards, fluctuating minimum pass scores and differences in state examinations, 
a single pass rate would not be a meaningful or useful assessment measure. The study found, 
however, that feedback from individual licensure examination boards a good measure of student 
achievement in specific disciplines and single program institutions (Korb, 1992). The Track 
Record Disclosure Law also called for reporting of licensure examination pass rates. This law 
which parallels the SRK Act, requires institutions who offer nc::'jaccalaureate vocational or 
technical programs to provide entering students with information pertaining to graduation, job 
placement and pass rates for licensure and certification exams (Fox, 1991). 

Legislative reporting requirements in many states now mandate that licensure examination 
pass rates be included as a component of their assessment report (see appendix 2). In contrast 
to the feasibility study by the SRK Act, these pass rates are by exam and not an single overall 
institutional pass rate. South Carolina and Tennessee requires not only the pass rate but the 
scores as well. 

Conclusion 

When the objective of a degree programs is preparation for a specific occupation, the 
outcomes measurement needs to measure occupational readiness. In addition, if the degree 
program encompasses a professional field, passing a licensure examination may be a prerequisite 



9 

ERIC 



9 

12 



for graduates before entering the occupation. Feedback from a man) licensure examinations is 
available to higher education institutions for the purposes of program improvement. When an 
good understanding of the scope, content, and limitations of a given examination exists, and it 
has been determined that the instrument mirrors the curriculum, the given licensure examination 
is a good outcomes measurement for that program. Even though no single measure can 
determine institutional or departmental quality, licensure exam results are a useful component 
of a comprehensive assessment process. 
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Appendix 2 

Current Status of State Legislative Reporting Requirements 
Occupational Licensure Examination Pass Rates 



State 

Florida 

Kentucky 

Lousianna 

South Carolina 

Tennessee 

West Virginia 

Oklahoma 

Arkansas 

Maryland 

North Carolina 

Texas 



Reporting Mandate 
Pass rates 
Pass rates 

Pass rates by institution 
Pass rates and scores 
Pass rates and scores 
Pass rates 

Optional for outcomes assessment report 
Not specifically called for 
Not specifically called for 
Not specifically called for 
Assessment plan under development 



Source; Bogue, G„ Creech, J. & Folger, J. (1993). Assessing Quality in Higher Education: 
Policy Actions in the SREB States. Atlanta: Southern Regional Education Board. 
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